Conversational Agents in Health Care: Expert Interviews to Inform the Definition, Classification, and Conceptual Framework

Background Conversational agents (CAs), or chatbots, are computer programs that simulate conversations with humans. The use of CAs in health care settings is recent and rapidly increasing, which often translates to poor reporting of the CA development and evaluation processes and unreliable research findings. We developed and published a conceptual framework, designing, developing, evaluating, and implementing a smartphone-delivered, rule-based conversational agent (DISCOVER), consisting of 3 iterative stages of CA design, development, and evaluation and implementation, complemented by 2 cross-cutting themes (user-centered design and data privacy and security). Objective This study aims to perform in-depth, semistructured interviews with multidisciplinary experts in health care CAs to share their views on the definition and classification of health care CAs and evaluate and validate the DISCOVER conceptual framework. Methods We conducted one-on-one semistructured interviews via Zoom (Zoom Video Communications) with 12 multidisciplinary CA experts using an interview guide based on our framework. The interviews were audio recorded, transcribed by the research team, and analyzed using thematic analysis. Results Following participants’ input, we defined CAs as digital interfaces that use natural language to engage in a synchronous dialogue using ≥1 communication modality, such as text, voice, images, or video. CAs were classified by 13 categories: response generation method, input and output modalities, CA purpose, deployment platform, CA development modality, appearance, length of interaction, type of CA-user interaction, dialogue initiation, communication style, CA personality, human support, and type of health care intervention. Experts considered that the conceptual framework could be adapted for artificial intelligence–based CAs. However, despite recent advances in artificial intelligence, including large language models, the technology is not able to ensure safety and reliability in health care settings. Finally, aligned with participants’ feedback, we present an updated iteration of the conceptual framework for health care conversational agents (CHAT) with key considerations for CA design, development, and evaluation and implementation, complemented by 3 cross-cutting themes: ethics, user involvement, and data privacy and security. Conclusions We present an expanded, validated CHAT and aim at guiding researchers from a variety of backgrounds and with different levels of expertise in the design, development, and evaluation and implementation of rule-based CAs in health care settings

Mobile digital education for health professions: Systematic review and meta-analysis by the Digital Health Education Collaboration

Background: There is a pressing need to implement efficient and cost-effective training to address the worldwide shortage of health professionals. Mobile digital education (mLearning) has been mooted as a potential solution to increase the delivery of health professions education as it offers the opportunity for wide access at low cost and flexibility with the portability of mobile devices. To better inform policy making, we need to determine the effectiveness of mLearning. Objective: The primary objective of this review was to evaluate the effectiveness of mLearning interventions for delivering health professions education in terms of learners’ knowledge, skills, attitudes, and satisfaction. Methods: We performed a systematic review of the effectiveness of mLearning in health professions education using standard Cochrane methodology. We searched 7 major bibliographic databases from January 1990 to August 2017 and included randomized controlled trials (RCTs) or cluster RCTs. Results: A total of 29 studies, including 3175 learners, met the inclusion criteria. A total of 25 studies were RCTs and 4 were cluster RCTs. Interventions comprised tablet or smartphone apps, personal digital assistants, basic mobile phones, iPods, and Moving Picture Experts Group-1 audio layer 3 player devices to deliver learning content. A total of 20 studies assessed knowledge (n=2469) and compared mLearning or blended learning to traditional learning or another form of digital education. The pooled estimate of studies favored mLearning over traditional learning for knowledge (standardized mean difference [SMD]=0.43, 95% CI 0.05-0.80, N=11 studies, low-quality evidence). There was no difference between blended learning and traditional learning for knowledge (SMD=0.20, 95% CI –0.47 to 0.86, N=6 studies, low-quality evidence). A total of 14 studies assessed skills (n=1097) and compared mLearning or blended learning to traditional learning or another form of digital education. The pooled estimate of studies favored mLearning (SMD=1.12, 95% CI 0.56-1.69, N=5 studies, moderate quality evidence) and blended learning (SMD=1.06, 95% CI 0.09-2.03, N=7 studies, low-quality evidence) over traditional learning for skills. A total of 5 and 4 studies assessed attitudes (n=440) and satisfaction (n=327), respectively, with inconclusive findings reported for each outcome. The risk of bias was judged as high in 16 studies. Conclusions: The evidence base suggests that mLearning is as effective as traditional learning or possibly more so. Although acknowledging the heterogeneity among the studies, this synthesis provides encouraging early evidence to strengthen efforts aimed at expanding health professions education using mobile devices in order to help tackle the global shortage of health professionals

Exploring views and experiences of the general public’s adoption of digital technologies for healthy lifestyle in Singapore: a qualitative study

ObjectiveLittle is known about the general adult population’s adoption of digital technology to support healthy lifestyle, especially when they are expected to take greater personal responsibility for managing their health and well-being today. The current qualitative study intended to gain an in-depth understanding of determinants of digital technology adoption for healthy lifestyle among community-dwelling adults in Singapore.DesignA qualitative study design, with thematic framework analysis was applied to develop themes from the data.SettingSemi-structured individual interviews were conducted with participants either face-to-face or online through a videoconferencing platform.Participants14 women and 16 men from the general population who were between the ages of 22 and 71 years.ResultsThree major themes were developed: (1) digitally disempowered (2) safety and perceived risks and harm; (3) cultural values and drives. Adoption of technology among the general population is needs-driven, and contingent on individual, technological and other cross-cultural contextual factors.ConclusionOur findings highlight there is no one solution which fits all individuals, emphasizing the challenges of catering to diverse groups to reduce barriers to adoption of digital technologies for healthy lifestyle. Digital guidance and training, as well as social influences, can motivate technological adoption in the population. However, technical problems as well as data security and privacy concerns should first be adequately addressed. This study provides rich cross-cultural insights and informs policy-making due to its alignment with government public health initiatives to promote healthy lifestyle

Designing, Developing, Evaluating, and Implementing a Smartphone-Delivered, Rule-Based Conversational Agent (DISCOVER): Development of a Conceptual Framework

Background: Conversational agents (CAs), also known as chatbots, are computer programs that simulate human conversations by using predetermined rule-based responses or artificial intelligence algorithms. They are increasingly used in health care, particularly via smartphones. There is, at present, no conceptual framework guiding the development of smartphone-based, rule-based CAs in health care. To fill this gap, we propose structured and tailored guidance for their design, development, evaluation, and implementation. Objective: The aim of this study was to develop a conceptual framework for the design, evaluation, and implementation of smartphone-delivered, rule-based, goal-oriented, and text-based CAs for health care. Methods: We followed the approach by Jabareen, which was based on the grounded theory method, to develop this conceptual framework. We performed 2 literature reviews focusing on health care CAs and conceptual frameworks for the development of mobile health interventions. We identified, named, categorized, integrated, and synthesized the information retrieved from the literature reviews to develop the conceptual framework. We then applied this framework by developing a CA and testing it in a feasibility study. Results: The Designing, Developing, Evaluating, and Implementing a Smartphone-Delivered, Rule-Based Conversational Agent (DISCOVER) conceptual framework includes 8 iterative steps grouped into 3 stages, as follows: design, comprising defining the goal, creating an identity, assembling the team, and selecting the delivery interface; development, including developing the content and building the conversation flow; and the evaluation and implementation of the CA. They were complemented by 2 cross-cutting considerations-user-centered design and privacy and security-that were relevant at all stages. This conceptual framework was successfully applied in the development of a CA to support lifestyle changes and prevent type 2 diabetes. Conclusions: Drawing on published evidence, the DISCOVER conceptual framework provides a step-by-step guide for developing rule-based, smartphone-delivered CAs. Further evaluation of this framework in diverse health care areas and settings and for a variety of users is needed to demonstrate its validity. Future research should aim to explore the use of CAs to deliver health care interventions, including behavior change and potential privacy and safety concerns. Keywords: chatbot; conceptual framework; conversational agent; digital health; mHealth; mobile health; mobile phone

Digital Education for Health Professions on Smoking Cessation Management:Systematic Review by the Digital Health Education Collaboration

BACKGROUND: Tobacco smoking, one of the leading causes of preventable death and disease, is associated with 7 million deaths every year. This is estimated to rise to more than 8 million deaths per year by 2030, with 80% occurring in low- and middle-income countries. Digital education, teaching, and learning using digital technologies have the potential to increase educational opportunities, supplement teaching activities, and decrease distance barriers in health professions education. OBJECTIVE: The primary objective of this systematic review was to evaluate the effectiveness of digital education compared with various controls in improving learners' knowledge, skills, attitudes, and satisfaction to deliver smoking cessation therapy. The secondary objectives were to assess patient-related outcomes, change in health professionals' practice or behavior, self-efficacy or self-rated competence of health professionals in delivering smoking cessation therapy, and cost-effectiveness of the interventions. METHODS: We searched 7 electronic databases and 2 trial registers for randomized controlled trials published between January 1990 and August 2017. We used gold standard Cochrane methods to select and extract data and appraise eligible studies. RESULTS: A total of 11 studies (number of participants, n=2684) were included in the review. All studies found that digital education was at least as effective as traditional or usual learning. There was some suggestion that blended education results in similar or greater improvements in knowledge (standardized mean difference, SMD=0.19, 95% CI -0.35 to 0.72), skill (SMD=0.58, 95% CI 0.08-1.08), and satisfaction (SMD=0.62, 95% CI 0.12-1.12) compared with digital education or usual learning alone. There was also some evidence for improved attitude (SMD=0.45, 95% CI 0.18-0.72) following digital education compared with usual learning. Only 1 study reported patient outcomes and the setup cost of blended education but did not compare outcomes among groups. There were insufficient data to investigate what components of the digital education interventions were associated with the greatest improvements in learning outcomes. CONCLUSIONS: The evidence suggests that digital education is at least as effective as usual learning in improving health professionals' knowledge and skill for delivering smoking cessation therapy. However, limitations in the evidence base mean that these conclusions should be interpreted with some caution. TRIAL REGISTRATION: PROSPERO CRD42016046815; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=46815

Exploring the Perceptions of mHealth Interventions for the Prevention of Common Mental Disorders in University Students in Singapore: Qualitative Study

Background: Mental health interventions delivered through mobile health (mHealth) technologies can increase the access to mental health services, especially among university students. The development of mHealth intervention is complex and needs to be context sensitive. There is currently limited evidence on the perceptions, needs, and barriers related to these interventions in the Southeast Asian context. Objective: This qualitative study aimed to explore the perception of university students and mental health supporters in Singapore about mental health services, campaigns, and mHealth interventions with a focus on conversational agent interventions for the prevention of common mental disorders such as anxiety and depression. Methods: We conducted 6 web-based focus group discussions with 30 university students and one-to-one web-based interviews with 11 mental health supporters consisting of faculty members tasked with student pastoral care, a mental health first aider, counselors, psychologists, a clinical psychologist, and a psychiatrist. The qualitative analysis followed a reflexive thematic analysis framework. Results: The following 6 main themes were identified: a healthy lifestyle as students, access to mental health services, the role of mental health promotion campaigns, preferred mHealth engagement features, factors that influence the adoption of mHealth interventions, and cultural relevance of mHealth interventions. The interpretation of our findings shows that students were reluctant to use mental health services because of the fear of stigma and a possible lack of confidentiality. Conclusions: Study participants viewed mHealth interventions for mental health as part of a blended intervention. They also felt that future mental health mHealth interventions should be more personalized and capable of managing adverse events such as suicidal ideation

LOWER URINARY TRACT SYMPTOMS AND DEPRESSION IN PATIENTS WITH MULTIPLE SCLEROSIS

Background: Both depression and lower urinary tract symptoms (LUTS) may be present in patients with multiple sclerosis (MS). The objective of this study was to give an insight on depression and LUTS in patients with MS in Croatia and to determine the possible association between LUTS and depression in patients with MS. Subjects and methods: This was a prospective cross-sectional study conducted in a tertiary healthcare center in Croatia. Hundred and one consecutive patients with MS (75 female, 26 male, mean age 42.09 (range 19-77) years, mean Expanded Disability Status Scale (EDSS) score 3.1 (range 0.0-7.0)) participated in this study. We evaluated LUTS and related quality of life (QoL) using three International Consultation on Incontinence Questionnaires (ICIQ) enquiring about overactive bladder (ICIQ-OAB), urinary incontinence short form (ICIQ-UI SF) and lower urinary tract symptoms related quality of life (ICIQLUTS-QoL). ICIQ-OAB and ICIQLUTS-QoL were for this purpose with permission successfully translated and validated into Croatian, while ICIQ-UI SF was already previously validated for the Croatian language. Information regarding treatment for depression was obtained during the medical interview. Data were analyzed and interpreted using IBM SPSS Statistics for Windows, version 23.0 (IBM Corp., Armonk, N.Y., USA). Results: 89.10% (N=90) patients with MS reported urgency with urge urinary incontinence (UUI) present in 70.29% (N=71). 81.18% (N=82) patients reported nocturia, and 90.09% (N=91) reported feeling drowsy or sleepy during the day due to bladder symptoms. Neurological deficit measured by EDSS was found to positively correlate with LUTS on all three questionnaires: ICIQOAB (r=0.390, p<0.05), ICIQ-UI SF (r=0.477, p<0.01) and ICIQ-LUTSQoL (r=0.317, p<0.05). 25 patients were in treatment for depression. There were no significant differences between female and male patients regarding treatment for depression (2=0.018, df=1, p>0.05). Results on ICIQ-UI SF showed that depressive patients had more pronounced LUTS (t=2.067, df=99, p<0.05), which was also true for the ICIQ-LUTSQoL (t=-2.193, df=99, p<0.05). Positive correlations were found between depression and LUTS on ICIQ-UI SF (r=0.203, p<0.05) and ICIQ-LUTSQoL (r=0.215, p<0.05). Conclusion: This study gives insight into the presence of depression and LUTS in Croatian patients with MS for which purpose ICIQ-OAB and ICIQ-LUTSQoL were with permission successfully translated and validated into Croatian. The connection between depression and LUTS must be considered when managing patients with MS

Serious gaming and gamification education in health professions: Systematic review

Background: There is a worldwide shortage of health workers, and this issue requires innovative education solutions. Serious gaming and gamification education have the potential to provide a quality, cost-effective, novel approach that is flexible, portable, and enjoyable and allow interaction with tutors and peers. Objective: The aim of this systematic review was to evaluate the effectiveness of serious gaming/gamification for health professions education compared with traditional learning, other types of digital education, or other serious gaming/gamification interventions in terms of patient outcomes, knowledge, skills, professional attitudes, and satisfaction (primary outcomes) as well as economic outcomes of education and adverse events (secondary outcomes). Methods: A comprehensive search of MEDLINE, EMBASE, Web of Knowledge, Educational Resources Information Centre, Cochrane Central Register of Controlled Trials, PsycINFO, and Cumulative Index to Nursing and Allied Health Literature was conducted from 1990 to August 2017. Randomized controlled trials (RCTs) and cluster RCTs were eligible for inclusion. Two reviewers independently searched, screened, and assessed the study quality and extracted data. A meta-analysis was not deemed appropriate due to the heterogeneity of populations, interventions, comparisons, and outcomes. Therefore, a narrative synthesis is presented. Results: A total of 27 RCTs and 3 cluster RCTs with 3634 participants were included. Two studies evaluated gamification interventions, and the remaining evaluated serious gaming interventions. One study reported a small statistically significant difference between serious gaming and digital education of primary care physicians in the time to control blood pressure in a subgroup of their patients already taking antihypertensive medications. There was evidence of a moderate-to-large magnitude of effect from five studies evaluating individually delivered interventions for objectively measured knowledge compared with traditional learning. There was also evidence of a small-to-large magnitude of effect from 10 studies for improved skills compared with traditional learning. Two and four studies suggested equivalence between interventions and controls for knowledge and skills, respectively. Evidence suggested that serious gaming was at least as effective as other digital education modalities for these outcomes. There was insufficient evidence to conclude whether one type of serious gaming/gamification intervention is more effective than any other. There was limited evidence for the effects of serious gaming/gamification on professional attitudes. Serious gaming/gamification may improve satisfaction, but the evidence was limited. Evidence was of low or very low quality for all outcomes. Quality of evidence was downgraded due to the imprecision, inconsistency, and limitations of the study. Conclusions: Serious gaming/gamification appears to be at least as effective as controls, and in many studies, more effective for improving knowledge, skills, and satisfaction. However, the available evidence is mostly of low quality and calls for further rigorous, theory-driven research

Just-in-Time Adaptive Mechanisms of Popular Mobile Apps for Individuals With Depression: Systematic App Search and Literature Review

BACKGROUND The number of smartphone apps that focus on the prevention, diagnosis, and treatment of depression is increasing. A promising approach to increase the effectiveness of the apps while reducing the individual's burden is the use of just-in-time adaptive intervention (JITAI) mechanisms. JITAIs are designed to improve the effectiveness of the intervention and reduce the burden on the person using the intervention by providing the right type of support at the right time. The right type of support and the right time are determined by measuring the state of vulnerability and the state of receptivity, respectively. OBJECTIVE The aim of this study is to systematically assess the use of JITAI mechanisms in popular apps for individuals with depression. METHODS We systematically searched for apps addressing depression in the Apple App Store and Google Play Store, as well as in curated lists from the Anxiety and Depression Association of America, the United Kingdom National Health Service, and the American Psychological Association in August 2020. The relevant apps were ranked according to the number of reviews (Apple App Store) or downloads (Google Play Store). For each app, 2 authors separately reviewed all publications concerning the app found within scientific databases (PubMed, Cochrane Register of Controlled Trials, PsycINFO, Google Scholar, IEEE Xplore, Web of Science, ACM Portal, and Science Direct), publications cited on the app's website, information on the app's website, and the app itself. All types of measurements (eg, open questions, closed questions, and device analytics) found in the apps were recorded and reviewed. RESULTS None of the 28 reviewed apps used JITAI mechanisms to tailor content to situations, states, or individuals. Of the 28 apps, 3 (11%) did not use any measurements, 20 (71%) exclusively used self-reports that were insufficient to leverage the full potential of the JITAIs, and the 5 (18%) apps using self-reports and passive measurements used them as progress or task indicators only. Although 34% (23/68) of the reviewed publications investigated the effectiveness of the apps and 21% (14/68) investigated their efficacy, no publication mentioned or evaluated JITAI mechanisms. CONCLUSIONS Promising JITAI mechanisms have not yet been translated into mainstream depression apps. Although the wide range of passive measurements available from smartphones were rarely used, self-reported outcomes were used by 71% (20/28) of the apps. However, in both cases, the measured outcomes were not used to tailor content and timing along a state of vulnerability or receptivity. Owing to this lack of tailoring to individual, state, or situation, we argue that the apps cannot be considered JITAIs. The lack of publications investigating whether JITAI mechanisms lead to an increase in the effectiveness or efficacy of the apps highlights the need for further research, especially in real-world apps

Medical doctors\u27 offline computer-assisted digital education: Systematic review by the digital health education collaboration

Background: The widening gap between innovations in the medical field and the dissemination of such information to doctors may affect the quality of care. Offline computer-based digital education (OCDE) may be a potential solution to overcoming the geographical, financial, and temporal obstacles faced by doctors.Objective: The objectives of this systematic review were to evaluate the effectiveness of OCDE compared with face-to-face learning, no intervention, or other types of digital learning for improving medical doctors\u27 knowledge, cognitive skills, and patient-related outcomes. Secondary objectives were to assess the cost-effectiveness (CE) of OCDE and any adverse effects.Methods: We searched major bibliographic databases from 1990 to August 2017 to identify relevant articles and followed the Cochrane methodology for systematic reviews of intervention.Results: Overall, 27 randomized controlled trials (RCTs), 1 cluster RCT (cRCT), and 1 quasi-RCT were included in this review. The total number of participants was 1690 in addition to the cRCT, which included 24 practices. Due to the heterogeneity of the participants, interventions, and outcomes, meta-analysis was not feasible, and the results were presented as narrative summary. Compared with face-to-face learning, the effect of OCDE on knowledge gain is uncertain (ratio of the means [RM] range 0.95-1.17; 8 studies, 495 participants; very low grade of evidence). From the same comparison, the effect of OCDE on cognitive skill gain is uncertain (RM range 0.1-0.9; 8 studies, 375 participants; very low grade of evidence). OCDE may have little or no effect on patients\u27 outcome compared with face-to-face education (2 studies, 62 participants; low grade of evidence). Compared with no intervention, OCDE may improve knowledge gain (RM range 1.36-0.98; 4 studies, 401 participants; low grade of evidence). From the same comparison, the effect of OCDE on cognitive skill gain is uncertain (RM range 1.1-1.15; 4 trials, 495 participants; very low grade of evidence). One cRCT, involving 24 practices, investigated patients\u27 outcome in this comparison and showed no difference between the 2 groups with low-grade evidence. Compared with text-based learning, the effect of OCDE on cognitive skills gain is uncertain (RM range 0.91-1.46; 3 trials with 4 interventions; 68 participants; very low-grade evidence). No study in this comparison investigated knowledge gain or patients\u27 outcomes. One study assessed the CE and showed that OCDE was cost-effective when compared with face-to-face learning if the cost is less than or equal to Can $200. No trial evaluated the adverse effect of OCDE.Conclusions: The effect of OCDE compared with other methods of education on medical doctors\u27 knowledge and cognitive skill gain is uncertain. OCDE may improve doctors\u27 knowledge compared with no intervention but its effect on doctors\u27 cognitive skills is uncertain. OCDE may have little or no effect in improving patients\u27 outcome